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LONG-TERM  GOALS 

The  verification  of  high-resolution  mesoscale  numerical  weather  predictions  presents  unique 
challenges.  Traditional  verification  metrics  -  root  mean  square  error,  etc.,  which  rely  on  single  point 
verification  often  give  incomplete  or  misleading  assessments  of  model  perfonnance.  Small-scale 
features  are  often  miss-represented  (aliasing)  or,  due  to  much  lower  predictability  than  large-scale 
features,  cause  an  unwarranted  penalty  by  conventional  verification  measures  due  to  small  spatial  or 
temporal  errors.  Both  the  model  developer  and  the  operational  user  need  better  metrics  in  order  to 
assess  the  performance  of  very  high-resolution  models.  Our  long-term  goal  is  to  contribute  to  better 
high-resolution  model  development  and  selection  by  developing  a  suite  of  verification  tools  to  assist 
both  the  model  developer  and  the  model  user. 

Towards  this  goal  we  are  developing  three  verification  techniques:  cluster  analysis  (Marzban  and 
Sandgathe,  2006,  2008a  and  Marzban,  Sandgathe,  and  Lyons,  2008);  variograms  (Marzban  and 
Sandgathe,  2008b);  and  optical  flow  (Marzban  and  Sandgathe,  2008c).  These  techniques  are  in  various 
stages  of  development  but  all  have  been  discussed  in  previous  annual  reports  and  tested  against  a  three 
model  data  set  of  three  different  WRF  formulations:  University  of  Oklahoma's  2km  WRF  (arw2), 
NCAR's  4km  WRF  (arw4),  and  NOAA's  Mesoscale  Model  (nmm4). 

We  are  now  refining  these  techniques  for  possible  transition  to  operational  usage  and  preparing  to 
demonstrate  them  on  a  year-long  data  set  of  the  Penn  State  Mesoscale  Model  (MM5);  the  NCAR 
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Weather  Research  and  Forecast  Model  (WRF)  and  the  Navy  Coupled  Ocean  and  Atmosphere 
Prediction  System  (COAMPS)  all  operating  at  the  University  of  Washington. 

OBJECTIVES 

The  initial  objective  of  this  research  effort  was  to  develop  the  variogram  approach  as  a  tool  for 
improving  our  other  mesoscale  verification  technique  (MVT),  namely  the  “box  size”  (or  characteristic 
length  scale  as  discussed  in  Hoffman  et  al  1995  and  Nehrkom  et  al.,  2003).  We  have  modified  that 
objective  to  explore  variograms  as  a  verification  tool  in  their  own  right.  We  are  still  pursuing  MVT; 
however,  we  have  now  shown  that  some  components  of  the  MVT  are  closely  related  to  what  is  known 
in  image  processing  circles  as  Optical  Flow  (OF).  OF  appears  to  lay  down  a  theoretical/mathematical 
foundation  for  MVT  which  adds  insight  into  the  inner- workings  of  MVT.  We  have  pursued  the  OF 
approach  to  verification  and  have  applied  it  to  the  above-mentioned  forecasts  (i.e.,  arw2,  arw4,  and 
nmm4). 

WORK  COMPLETED 

At  last  report,  we  had  developed  all  three  verification  methods  at  a  level  of  maturity  that  warranted 
their  application  to  a  large  data  set  consisting  of  32  days  of  obervation  and  forecasts  from  three 
different  NWP  formulations.  In  doing  so,  it  became  clear  that  the  variogram  method  must  be  extended 
to  allow  for  the  computation  of  variograms  for  only  nonzero  grid  values.  The  task  was  nontrivial, 
because  such  grid  values  constitute  an  irregular  grid.  That  work  is  now  complete.  The  value  of  such  a 
variogram  is  in  its  ability  to  capture  only  intensity  errors.  By  contrast,  the  original  variogram, 
computed  across  the  entire  grid,  assesses  intensity  and  phase  errors  (size  and  displacement).  In  fact,  in 
the  application  of  the  method  to  large  data  sets,  we  are  computing  both  type  of  variograms  -  one  for 
assessing  forecast  quality  in  terms  of  total  error  (phase  and  intensity),  and  another  for  intensity  error 
only.  All  of  this  has  been  described  in  a  revised  version  of  an  article  (Marzban  and  Sandgathe  2008b), 
which  is  now  conditionally  accepted  for  publication,  pending  minor  revision. 

As  for  the  optical  flow  method,  it  has  been  applied  to  4  days  consisting  of  large  winter  stonn  events. 
Only  sea-level  pressure  is  examined  thus  far,  because  the  continuity  of  the  field  is  more  consistent  with 
the  assumptions  underlying  the  method.  The  approach  leads  to  two  quantities  -  1)  a  2d  field 
displaying  the  vectors  that  best  map  the  observed  to  the  forecast  field  (i.e.,  called  the  OF  field),  and  2) 
a  joint  histogram  of  the  magnitude  and  direction  of  the  vectors  across  an  OF  field.  The  former 
provides  a  visual  display  of  the  quanlity  of  the  forecasts,  while  the  latter  offers  a  summary  of  the 
comparison.  For  example,  a  peak  in  the  joint  histogram  would  suggest  a  coherent  relative  shift  of  the 
two  fields.  The  methodology  has  been  extended  to  allow  the  averaging  of  the  OF  field  across  multiple 
days.  Details  of  the  approach  appear  in  article  which  is  conditionally  accepted  for  publication,  pending 
minor  revision  (Marzban  and  Sandgathe  2008c). 

RESULTS 

Figure  1  shows  both  variograms  -  across  entire  field  (top),  and  across  only  nonzero  grid  values 
(bottom)  -  of  the  observed  field  (black),  arw2  (red),  arw4  (green),  and  nmm4  (blue),  all  for  May  13, 
2005.  From  the  left  panel  it  can  be  seen  that  on  shorter  scales  (e.g.,  less  than  2000Km),  the  3  models 
have  variograms  that  mostly  agree  with  that  of  observed  field,  at  least  after  taking  into  account  the 
error-bars.  On  longer  scales  (exceeding  2000km),  although  arw2  and  arw4  agree  with  the  observed 
variogram,  nmm4  is  significantly  higher.  This  suggests  that  on  larger  scales,  nmm4  does  not  “de- 
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correlate”  as  quickly  (as  a  function  of  scale)  as  arw2,  arw4,  and  the  observations.  The  right  panel 
assesses  only  intensity  errors,  and  tells  a  different  story:  Most  notably,  even  on  smaller  scales  the 
variogram  for  nmm4  under  estimates  that  of  the  observed  field.  arw2  and  arw4  continue  to  provide 
reasonable  estimates  of  the  observed  variogram;  the  exception  is  on  larger  scales,  where  none  of  the 
models  properly  represent  the  observed  variogram. 

The  comparison  of  the  variograms  for  the  observed  and  forecasts  fields  (across  entire  field)  for  all  32 
days  in  the  data,  suggests  that  arw2  and  arw4  have  reasonable  forecasts  across  all  scales,  but  nmm4  is 
inconsistent  with  observed  variograms  across  all  scales.  However,  in  terms  of  intensity  error  alone 
(i.e.,  using  variograms  computed  across  only  nonzero  grid  values),  it  turns  out  that  all  three 
formulations  are  statistically  equivalent  with  each  other  and  equally  consistent  with  observations.  The 
models  produce  variograms  that  are  slightly  lower  than  observed,  but  the  difference  is  not  statistically 
significant. 

Figure  2  shows  the  OF  field  relating  the  forecast  sea-level  pressure  to  that  observed  for  Dec.  4,  2007. 
Whereas  a  visual  comparison  of  the  forecast  and  observed  fields  (top  row)  does  not  immediately 
highlight  areas  of  disagreement,  the  OF  field  (bottom  row)  clearly  shows  the  differences.  Such  OF 
fields  and  their  joint  histograms  have  been  computed  for  4  days  (not  shown),  and  the  average  OF  field 
across  the  4  days  is  shown  in  Figure  3.  The  spatial  distribution  of  the  errors  is  clearly  highlighted. 

We  are  participating  in  an  international  verification  exercise  hosted  by  the  National  Center  for 
Atmospheric  Research  (NCAR).  They  have  established  a  set  of  idealized  cases  and  also  a  standardized 
data  set  of  5  days  observations/forecasts  from  the  2005  Spring  Experiment.  We  have  tested  our  three 
techniques  on  this  comprehensive  data  set  for  comparison  with  other  mesoscale  verification 
techniques.  In  addition,  we  have  started  using  the  University  of  Washington  Short  Range  Ensemble 
Forecast  (SREF)  system  (MM5  and  WRF)  in  combination  with  data  from  the  APL-UW  COAMPS-OS 
prediction  system.  We  presented  a  verification  using  both  variograms  and  OF  of  four  Northwest  winter 
storm  events  at  the  March  Pacific  Northwest  Weather  Workshop. 

In  March,  we  began  archiving  UW  SREF  MM5  and  WRF  data  and  APL  COAMPS-OS  data  in  order  to 
process  a  large  data  set  using  our  three  verification  techniques.  We  are  currently  testing  the  three 
techniques  on  all  three  model  formulations  on  at  least  eight  months  of  continuous  forecasts. 
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Figure  1:  Variogram  of  the  observed  reflectivity  field  on  May  13,  2005  (black),  and  the  correspondin 
g  24hr  forecasts  from  arw2  (red),  arw4  (green),  and  nmm4  (blue).  Top  variogram  is  computed 
across  entire  field,  while  bottom  variogram  excludes  zero  grid  values. 
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Figure  2:  Sea-level  pressure  forecast  (top,  left)  and  observed  (top,  right)  fields  for  Dec.  3,  2007. 
Also  shown  are  the  optical  flow  field  superimposed  on  the  two  fields  (bottom  row). 


Figure  3:  The  optical  flow  field  for  sea-level  pressure  averaged  over  4  days 
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CURRENT/FUTURE  WORK 


In  regards  to  the  COAMPS-OS  system,  Figure  5  is  an  example  of  a  twelve-hour  forecast  output  field  at 
1 5km  resolution. 


NRL  COAMPS-OS®  P_S_short  15.0km 
Valid  Time:  00:00Z  09  2007  OCT  Forecast:  12:00 
Base  Time:  12:00Z  08  OCT  2007 
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Figure  5.  COAMPS-OS  12  hour  forecast  of  air  temperature,  MSL  sea  level  pressure  and  10m  winds. 
The  domain  is  centered  over  the  northwest  of  the  United  States. 


RELATED  PROJECTS 

A  parallel  effort  is  a  verification  techniques  based  on  statistical  cluster  analysis.  It  is  funded  by  NSF 
for  rapid  transition  into  the  DoD/NOAA/NCAR  mesoscale  WRF  Developmental  Testbed  Center 
(DTC).  The  funding  completed  in  August,  2008.  It  has  been  applied  to  precipitation  fields  (Marzban 
and  Sandgathe,  2006),  a  variation  on  that  theme  has  been  applied  to  reflectivity  fields  (Marzban  and 
Sandgathe,  2008a),  and  an  improved  version  (that  allows  the  incorporation  of  meteorological 
parameters  into  the  analysis)  has  been  reported  by  Marzban,  Sandgathe,  Lyons  (2008). 
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